Data storage systems use specialized hardware and software along with storage media (e.g., disk, tape drives or flash memory) to provide fast and reliable storage for computing and data processing applications. Traditional storage systems rely on static definitions of system resources, such as host and storage IDs. Virtualization of storage helps achieve location independence by abstracting the physical location of the data. The virtualization system provides a logical space for data storage and the logical space to the actual physical location, but also adds a dynamic element to the definitions of system resources, as storage addresses are no longer statically tied to physical memory locations.
The Unix file system (UFS) has become the standard file system for most modern data storage systems, and advanced storage systems use updated variants such as the 64-bit UFS (UFS64). The UFS64 file system provides various advantages such as the ability to scale in size while providing capacity efficiencies and robust error-recovery mechanisms. Other file systems used by current data storage systems include the Common Block File System (CBFS), which creates virtual devices that emulate physical storage and presents data to users as files and folders of a local virtual disk. High capacity file systems, and especially those that implement or facilitate virtualization present certain file management issues, such as file system fragmentation, which is caused by non-contiguous or non-sequential layout of related data on the physical storage devices. Thus, the introduction of the advanced storage virtualization techniques like thin provisioning, multi-level stacking of the file systems, write anywhere snapshots, and de-duplication results in non-linear and disjoint layout of the volumes, file system, files and objects hosted by the file system. The non-contiguous nature of the volumes, file system/file/objects leads to address space fragmentation, which in turn leads to poor system performance. In particular, these issues impact the latency and the effective throughput of the system for certain types of workloads.
Techniques for performing fragmentation detection in file systems are relatively common in the storage world and there are many proven ways to detect and address the fragmentation of file system and files. Current approaches, however, are tied to specific storage system architectures, and most fragmentation detection algorithms depend on static layout analysis to determine the fragmentation. Thus, although other systems may support fragmentation detection and defragmentation, they consider only the static layout of the file system and files and do not take into consideration the fact that virtualization in the storage brings many order of indirections, and characteristics such as media specific performance sensitive layout policies, cache density, cache efficiency, the nature of the workload, and so on. In addition, no practical fragmentation detection solution exists for UFS64, CBFS and other advanced file systems.
What is needed, therefore, is a system and method of detecting fragmentation by taking many variants of the end-to-end architecture of the storage system into consideration, as opposed to simply the static layout of the system. What is further needed is a system that takes into consideration system and application-specific characteristics, such as the nature of the workload, the effect of multiple layers of virtualization, cache density, spread of the IO (input/output) on the backend spindles, nature of the media, and so on.
The subject matter discussed in the background section should not be assumed to be prior art merely as a result of its mention in the background section. Similarly, a problem mentioned in the background section or associated with the subject matter of the background section should not be assumed to have been previously recognized in the prior art. The subject matter in the background section merely represents different approaches, which in and of themselves may also be inventions. EMC and VNX are trademarks of EMC Corporation.